Papers with sparse activation
Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on how large language models process multiple languages, but internal mechanisms of LLMs remain insufficiently explored. |
| Approach: | They propose to convert dense LLMs into fine-grained MoE architectures and analyze their activation patterns using expert activation frequency heatmaps. |
| Outcome: | The proposed method outperforms random expert pruning and exceeds models in some languages. |
From Pseudo-Balancing to True Specialization: Memory-Aware Routing for Mixture-of-Experts (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to optimize expert-centered load balancing fail to account for pseudo-balance phenomenon . severe knowledge overlap among experts leads to redundant representations and inefficient parameter utilization . |
| Approach: | They propose a method that prioritizes expert utilization over semantic alignment . they use memory-aware routing to ensure expert load balancing is consistent . |
| Outcome: | Experimental results show that MAR improves expert specialization by 35% and accuracy by 2%-25% . MAR matches baseline performance with only half the experts . |